Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update hashes of all components to latest versions #233

Merged
merged 4 commits into from
Apr 5, 2022

Conversation

mkavulich
Copy link
Collaborator

DESCRIPTION OF CHANGES:

This is a badly needed update to the hashes of the UFS SRW components. Some of the current hashes are well out-of-date at this point, and missing several critical fixes that have been added to various components.

regional_workflow: 405553b (Mar 22) --> 3f83cde (Apr 4)
UFS_UTILS: f30740e (Nov 5 2021!) --> 31271f7 (Mar 25)
ufs-weather-model: e593349 (Feb 23) --> 96dffa1 (Apr 4)
UPP: 4a16052 (Feb 10) --> 394917e (Apr 4)

TESTS CONDUCTED:

Per comments on regional_workflow issue #673, nearly all WE2E tests pass now on Hera (way more than the current set of hashes). I believe this is enough to be confident in updating the hashes.

DEPENDENCIES:

None.

DOCUMENTATION:

If this PR is contributing new capabilities that need to be documented, please also include updates to the RST files (docs/UsersGuide/source) as supporting material.

@mkavulich
Copy link
Collaborator Author

I decided to run a build test on a few other platforms, and it's a good thing! There are a few other build issues that needed to be fixed:

  1. The latest UPP requires CMake 3.19+, and Cheyenne and Orion were both using 3.18 versions. I have now updated the CMake versions for those two platforms.
  2. The latest UFS_UTILS requires g2 v3.4.3. This led me down a horrible rabbit hole, because we did not load this version of g2 in srw_common, yet it was being used on Hera (and Jet, and Orion as it happens). This is because of a known issue with hpc-stack modulefiles that causes some modules to "overwrite" the selected module version under the right conditions (see that issue for more details); this caused my tests to be successful despite the build files being objectively wrong. I have updated srw_common to include g2 v3.4.3, and included a temporary "hack" to get things building on Cheyenne (the g2 v3.4.3 module must explicitly be loaded a second time, because this hpc-stack bug loads the default g2 v3.4.2 despite explicitly asking for v3.4.3). Hopefully fixes with hpc-stack on disk will remove the necessity for this hack in the future.

@mkavulich mkavulich merged commit 2b09220 into ufs-community:develop Apr 5, 2022
@mkavulich mkavulich deleted the feature/update_hashes branch April 20, 2022 16:39
christinaholtNOAA referenced this pull request in NOAA-GSL/ufs-srweather-app Jun 7, 2022
* Add support on NSSL/Odin (#227)

* Add support on NSSL/Odin

* Add wlfow_odin.env and modify detect_machine.sh

* update comment

* Detect explicitly odin1/odin2

* Add python module to cheyenne build environments (#232)

* Update SRW Documentation (#212)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

Co-authored-by: gspetro <[email protected]>

* Update hashes of all components to latest versions (#233)

* Add a Contributor's Guide feature to the docs (#228)

* create contributor's guide

* add guidelines for making good PR

* good pull request edits

* 1st Draft of COntributor's Guide

* minor formatting edits

* Add instructions for srw_common module load

* fix module files guidance

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* add info on documentation requirements

* 1st round of J. Beck's edits

* add textonly issue template + minor typo-type fixes

* minor edits/formatting

* remove pylintrc requirement

* fixed .yaml & other minor

Co-authored-by: gspetro <[email protected]>

* Updates to parameters in config_defaults .rst files (#237)

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

Co-authored-by: gspetro <[email protected]>

* Added functionality for MacOS X (#242)

* Added functionality for MacOS X

 Functionality for MacOS, updated module list in srw_common

* Update build_macos_gnu.env

* Update srw_common

* Update build_macos_gnu.env

The env/build_mac_gnu.env does not load srw_common module, but instead loads individual HPC-stack modules built locally on the Mac that contain higher-versions of some packages. This avoids conflicts with SRW builds for other platforms.

* Update srw_common

corrected the version of the gftl-shared

* Update build_macos_gnu.env

No need to load libpng module found in srw_common, as this is not being built as a part of the hpc-stack on MacOS X, rather installed system-wide.

Co-authored-by: Natalie Perlin <[email protected]>

* Add gaea to supported platforms (#236)

* fixes for gaea

* updates for gaea

* tweak for build env

* 2nd tweak for build env

* Fixes for slurm

* another fix for env

* added version for cmake

* Update Externals.cfg

* Update wflow_gaea.env

* tweak

* pulling Externals.cfg explicitly from develop

* temporarily removing externals

* Checked out directly from develop

* Bug fix with singularity env files (#245)

* Replace bash env files with modules  (#238)

* Pass machine name to build scripts.

* Use modules environment instead of shell scripts.

* Leave conda activation to the user.

* Remove set_machine script.

* Rename env to modulefiles

* Minor fix.

* Minor fix

* Take out *module purge* from modufiles and put it in devbuild.sh

* Activate conda directly in signularity modulefile.

* Minor fixes.

* Add Gaea modulefiles.

* Restore odin env files.

* Bug fixes in singularity modulefiles.

* Move activation of Lmod to devbuild.sh

* Don't do 'module purge' on cray systems

* Put Lmod initialization code in separate script.

* Go back to using modulefile for odin.

* Optionally pass machine name to lmod-setup.sh

* Modify odin wflow modulefile.

* Allow unknown platforms in devbuild.sh

* Update documentation.

* Move cmake init out of lmod-setup.sh on odin

* Also update markup language build documentation.

* Lmod setup script for both bash and tcsh login shells.

* Some fixes for tcsh login shell.

* Add singularity platform to lmod-setup

* update hash of regional workflow (#247)

* Update WE2E documentation (#241)

## DESCRIPTION OF CHANGES: 
This updates the documentation for how to use the WE2E testing system.

## TESTS CONDUCTED: 
Compiled the documentation using `sphinx-build` and viewed the resulting html.

## DEPENDENCIES:
PR #[745](ufs-community/regional_workflow#745) in regional_workflow.

## CONTRIBUTORS: 
@gspetro

* fixes for gaea modules (#248)

* Update regional_workflow hash, add shortcuts for common devbuild.sh options (#251)

* Modifications to `CODEOWNERS` file (#252)

## DESCRIPTION OF CHANGES: 
Make the following modifications to the github `CODEOWNERS` file:
1) Add several EPIC staff (@gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA) so they are notified of all PRs and can review them.
2) Remove duplicate entries.
3) Remove users who will no longer be working with the repo (thus far only @jwolff-ncar) .

## TESTS CONDUCTED: 
None.

* Add verification tasks to documentation (#243)


* METplus, model config & forecast params, separator

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* condense VX task table using ##

* update README

* add png and revert hpc-stack commits until PR#240 (mac docs) is approved

* jwolff edits

* add info on run_vx.local

Co-authored-by: gspetro <[email protected]>

* Add NOAA cloud platforms to SRW (#221)

* updates for noaacloud

* working version

* pointing to noaa-epic for testing

* changes for noaacloud

* switched to load-any

* fix for regional_workflow pointer (#260)

* update hash of regional workflow (#261)

* Feature/cheyenne fix (#258)

* Adding a github action to build on cheyenne with intel

* fixing yml

* fixes for missing load-any on cheyenne

* added pio as well

* Update .github/workflows/build.yml

Co-authored-by: Will Mayfield <[email protected]>

Co-authored-by: Will Mayfield <[email protected]>

* tweaks for build/run on gaea (#254)

* tweaks for build/run on gaea

* fixed path

* Updates for PR

* Check-in Jenkinsfile and unified scripts (#253)

* Add Jenkinsfile that includes a build and test pipeline, which
leverages the unified build and test scripts
* Supported platforms are Cheyenne, Gaea, and Orion
* Supported compilers are GNU and Intel

* Fix for miniconda3 load on Hera (#257)

Pin down the version of miniconda3 on Hera, and do not append to the module path

* Updates to Remaining Chapters (6 & 8-12) of SRW Docs/User's Guide (#255)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

Co-authored-by: gspetro <[email protected]>

* Update compiler prerequisite in docs (#267)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

* clarify compiler prereqs

Co-authored-by: gspetro <[email protected]>

Co-authored-by: Yunheng Wang <[email protected]>
Co-authored-by: Will Mayfield <[email protected]>
Co-authored-by: Gillian Petro <[email protected]>
Co-authored-by: gspetro <[email protected]>
Co-authored-by: Michael Kavulich <[email protected]>
Co-authored-by: Natalie Perlin <[email protected]>
Co-authored-by: Natalie Perlin <[email protected]>
Co-authored-by: Mark Potts <[email protected]>
Co-authored-by: danielabdi-noaa <[email protected]>
Co-authored-by: Chan-Hoo.Jeon-NOAA <[email protected]>
Co-authored-by: gsketefian <[email protected]>
Co-authored-by: Jesse McFarland <[email protected]>
SamuelTrahanNOAA pushed a commit to SamuelTrahanNOAA/ufs-srweather-app that referenced this pull request Sep 15, 2023
* UPdate hasd for GSI, regional workflow and model.

* Update hash for GSI and regional workflow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants